The Thing That Makes It Can Never Be The Thing That Grades It
Ask an AI to write something, then ask the same AI whether it is any good, and you already know the answer. It is good. It is always good. That is not dishonesty, it is structure — you handed one system both jobs and it has no way to be surprised by its own work.
The card this post is built on calls the fix loop engineering, and it opens with the line that does all the work: the AI writes the code, then decides the code is good. Nobody else looked. Six rules follow, but five of them are plumbing. The first rule is the whole idea, and it is one sentence long: the thing that makes it can never be the thing that grades it.
What makes the card worth a full article rather than a quick save is that this is not really about code. It is a description of how anything improves when you cannot trust the person doing the work to mark it — which is every creative job, including yours. So this article does two things: sets out the six rules as the card gives them, and then translates the same loop to the work of running a channel, where the maker and the checker are usually the same tired person at eleven at night.
The loop, as four steps
Down the middle of the card runs a spine with four labels on it. That spine is the actual machine. Everything else on the card is a rule about how to build one of the four parts properly.
| 1 · EDIT | 2 · RUN & LOG | 3 · VERIFY | 4 · KEEP OR RESET |
|---|---|---|---|
| The agent changes one file, and only that file | A short run produces a number, and the row is written down | Something outside the agent decides pass or fail | A better score is committed; a worse one is reverted at once |
| Everything else is read-only | Hypothesis, result and lesson, every cycle | A test suite, a benchmark, a rubric, or a script that exits 0 or 1 | Every win becomes the new floor |
| Without it the agent edits the marking scheme | Without it cycle fifty repeats cycle three | Without it you are back to marking your own homework | Without it the score wanders instead of climbing |
The six rules as the card gives them
Below is the card in full — the six panels, each with its headline, its one-line instruction, and the reason it gives. The wording is the card’s own.
| Panel | Headline | What it says | The instruction | Why |
|---|---|---|---|---|
| 1 · The one rule | Split maker and checker | The doer and the checker must be completely separate | Add a reviewer that can never edit tests | If it can edit the test, it makes the test easier, not the work better |
| 2 · The files | Keep it to three files | One file the agent edits, one read-only evaluation, one file of your instructions | Edit the working file only, never the evaluation | That is the whole architecture. Nothing else to build |
| 3 · The memory | Log every experiment | Every cycle logs the hypothesis, the result and the lesson | Log the commit, the metric, the status and the idea | By cycle fifty it decides with forty-nine experiments behind it |
| 4 · The verifier | Build a strong verifier | A test suite, a benchmark, a rubric, or a script that exits 0 or 1 | Gate every change on a clean pass | Weak verifier, fast garbage. Strong verifier, real progress |
| 5 · The ratchet | Only let the score rise | If the metric gets worse after a short run, the change is reverted instantly | Revert any run that scores worse | Every win becomes the new floor |
| 6 · The payoff | Let it check 100 times | Around 700 experiments over two days, about 20 kept wins | Build the loop, not the answer | Not smarter. It checked a hundred times, you check once |
WATCH THE ORIGINAL — 45 SECONDS
All six panels, the loop drawn down the middle, and the numbers from the run at the bottom.
The result the card is built on
The numbers at the foot of the card come from a run by Andrej Karpathy on nanochat — code he wrote and had already tuned himself. Around 700 experiments over roughly two days. About 20 kept wins. Training reached the same quality bar around 11% faster, with the time figure moving from roughly 2.02 hours to 1.80 hours. Along the way the loop found a missing normalisation scaler in his own code that he had walked past for months.
The sentence under those numbers is the one to remember: not smarter — it checked a hundred times, you check once. Nothing in that run required a model that reasons better than a person. It required a setup where a weak idea gets thrown away in five minutes without anybody’s feelings being involved, seven hundred times in a row.
Two things are worth saying plainly about those figures. They are one person’s run on one codebase, reported by him, and an eleven per cent improvement on a training time is a real result but not a transformation. And the loop only produced it because the metric was a number a script could read. Take away the number and the whole apparatus has nothing to ratchet against.
Why the first rule is the only one that matters
Rules two to five are engineering. Rule one is the reason they work, and it is the one people quietly drop first.
If the same agent can edit both the work and the test, improving the score has two solutions. It can do the hard thing, or it can loosen the test. Given a thousand attempts and no supervision, a system optimising a number will find the cheap path, not because it is cunning but because the cheap path is genuinely a path. The card’s own phrasing is exact: if it can edit the test, it makes the test easier, not the work better.
Which makes this a permissions problem before it is a prompting problem. The reviewer should be unable, not merely unwilling, to touch the evaluation. That is the same instinct we argued for when handing tools access to your accounts — start with the narrowest permission that works and widen only when something genuinely fails, which is the case we made in Read-Only First.
The same loop, pointed at a channel
Here is where this stops being a coding post. Every rule on that card describes a failure that creators already have, with different nouns.
You write the title. You decide the title is good. Nobody else looked. You cut the first thirty seconds, you watch it back, and of course it flows — you know what is coming next, so you cannot experience the confusion a stranger would feel. You are the maker and the checker, at eleven at night, on the same piece of work. That is the entire problem the card describes, and no amount of trying harder fixes a structural conflict.
The translation is not complicated. Your verifier is the number you agreed on before you made the thing: the retention curve at thirty seconds, the click-through on the thumbnail after a fixed number of impressions, the answer a person who has never seen it gives to one specific question. Your memory is a single file where every title you tried, its result and the lesson go in one row. Your ratchet is that the new version has to beat the old number or it goes back.
| The rule | What it looks like on a channel | The channel version | The way people get it wrong |
|---|---|---|---|
| Split maker and checker | You wrote the hook and you decided it lands | Ask someone who has not seen it: “what do you think this video is about?” | Asking a friend who wants you to feel good |
| Keep it to three files | Notes scattered across five apps and a notebook | One document of rules, one working draft, one results file | Rewriting the rules every time a video underperforms |
| Log every experiment | You remember the winners and forget the rest | One row per attempt: what you changed, the number, the lesson | A log with no lesson column, which is just a spreadsheet |
| Build a strong verifier | “It feels better” | A number chosen before publishing, read the same way every time | Picking the metric after you see the result |
| Only let the score rise | A redesign that quietly performs worse and stays | If the number drops, go back to the previous version | Changing three things at once so nothing is attributable |
| Let it check 100 times | One idea, published, hoped over | Twenty cheap tests on titles and thumbnails before one upload | Treating every test as a launch |
Four honest caveats
A loop needs a number, and most creative work does not have one. This method is powerful exactly where a script can read the score. Whether a video is good is not that. Whether a thumbnail gets clicked is. Be strict about which of the two you are actually optimising, because a loop pointed at a proxy will happily improve the proxy and damage the thing.
Seven hundred experiments are not free. Every cycle is a model call and a run, and the bill arrives whether the idea was good or not. The costs that hide behind an open-source or self-hosted setup are the subject of Open-Source Agents Still Pay For Models.
The numbers are one person’s run. Roughly 700 experiments, about 20 kept wins, around 11% faster on his own codebase over two days, as reported by him. That is a genuine result on a well-tuned project, not a benchmark, and not a promise about yours.
Separation has to be structural, not stated. Telling one agent to “act as an independent reviewer” of its own output is theatre. It is the same model, with the same blind spots, holding the same context. Separation means a different process, a different prompt, and above all no ability to change what it is grading against.
How to build the smallest version of this today
You do not need an agent framework to get most of the value. Pick one thing you keep redoing. Write down, before you start, the single number that says whether a version is better. Make a file with three columns — what you changed, what the number was, what you learned. Then make a rule you will actually keep: if the number goes down, the change goes back.
That is four of the six rules, and it takes an afternoon. The fifth is the one that costs something: find a checker that is not you. A person who has not seen the work, a tool that scores it against a rubric you wrote in advance, or a second model given only the output and the rubric — never the draft, never the ability to rewrite the rubric.
If you want a starting point for the rubric itself, the approach of taking something that already worked and reducing it to its structure is in Reverse-Engineer A Viral Post. And for keeping your tooling deliberately small while you do it, Pick One Per Stage And Learn It Properly.
Questions people ask
Can I just use two different models as maker and checker? It helps, and it is better than one model doing both. But the important separation is not which model — it is that the checker cannot modify the thing it grades against.
What counts as a strong verifier if I am not writing code? Anything that gives the same answer twice on the same input. A rubric with numbered criteria written before you looked at the work. A single question put to a stranger. A metric with a stated threshold. What disqualifies a verifier is being able to argue with it after the fact.
Why log the lesson and not just the result? Because the result stops being useful once you have moved on, and the lesson is what the next cycle reads. A log of numbers tells you what happened. A log with a lesson column tells the next attempt what not to try again.
Is reverting really better than fixing? For a loop, yes. Reverting is instant and certain; fixing is a second experiment wearing the clothes of the first. Revert, write the lesson down, and let the fix be its own cycle with its own row.
Build the loop, not the answer
The line printed along the bottom edge of the card is the best summary of it: build the loop, not the answer. Most of us spend our effort trying to get the right output on the first attempt, from ourselves or from a model. The alternative is to spend that effort once, on the machinery that lets a hundred wrong attempts be discarded cheaply — and then to let it run while you do something else.
And keep rule one intact, because everything rests on it. The thing that makes it can never be the thing that grades it. That is true of agents, and it was true of you long before any of this existed.
Learning a lot does not make anyone good. Doing a lot does.
If you want a structure that forces the doing, we run a free challenge — there is no barrier to entry, only a commitment fee that is refunded when you finish. It is, in its own small way, a verifier you cannot argue with.
What is your checker — a person, a number, or nobody at all? Tell us in the comments under the original reel.
More on building a channel for a global audience at mmoyoutube.com.
6 Comments